Method and Apparatus for Training a Probe Model Based Machine Vision System

ABSTRACT

A method for training a pattern recognition algorithm including the steps of identifying the known location of the pattern that includes repeating elements within a fine resolution image, using the fine resolution image to train a model associated with the fine image, using the model to examine the fine image resolution image to generate a score space, examining the score space to identify a repeating pattern frequency, using a coarse image that is coarser than the finest image resolution image to train a model associated with the coarse image, using the model associated with the coarse image to examine the coarse image thereby generating a location error, comparing the location error to the repeating pattern frequency and determining if the coarse image resolution is suitable for locating the pattern within a fraction of one pitch of the repeating elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/249,318 which was filed on Oct. 10, 2008 and which is titled “Methodand Apparatus for Training a Probe Model Based Machine Vision System”.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

BACKGROUND OF THE INVENTION

The present invention is related to training algorithms for machinevision systems and more specifically to an iterative process thatgenerates an optimal machine vision process for recognizing a pattern inimages.

One task performed by many machine vision systems is to attempt toidentify the location and orientation of a pattern of interest withinimages. Several different classes of algorithms have been developed toperform this pattern locating task.

One advantageous class of such algorithms uses a model to represent thepattern of interest which includes a plurality of probes. Each probe isa measure of similarity of a run-time image feature or region to apattern feature or region at a specific location. The plurality ofprobes is applied at a plurality of poses to the run-time image and theinformation from the probes at each pose is used to determine the mostlikely poses of the pattern in the run-time image. The individual probesmay be as described in U.S. Pat. No. 7,016,539 which is titled “MethodFor Fast, Robust, Multi-dimensional Pattern Recognition” which issued onMar. 21, 2006 and which is incorporated herein by reference in itsentirety, or could be something different, for example normalizedcorrelation. Hereafter this class of pattern matching algorithms will bereferred to as probe model algorithms.

To speed up the pattern recognition process, at least some probe modelalgorithms use a set of models corresponding to different imageresolutions to iteratively determine the pose of a pattern in an image.For instance, an algorithm may include five different models where eachmodel is for a different resolution of image. Here, during the patternrecognition process, a coarse resolution and associated model may beused initially to identify a coarse approximated pose of an instance ofa pattern in an image. Thereafter, a relatively finer resolution may beused to more precisely identify the pose of the pattern instance in theimage. This iterative process continues until a finest resolution modelis used and the precise pose of the pattern instance is identified.

For probe model algorithms to work properly, robust models need to bedeveloped during a model training process for each pattern to berecognized within an image. Known model training algorithms have severalshortcomings. For instance, in some cases a training algorithm willselect an initial image resolution for subsequent recognition processesthat may not work properly. In another instance the user may manuallyenter an initial image resolution that may also not work properly. Tothis end, in lower resolution images, pattern features that aredetectable when a pattern is aligned in one juxtaposition with respectto a camera may disappear when the pattern is aligned slightlydifferently with the camera due to sub-pixel phase shift. Where an imageresolution is selected during a training process with a camera alignedin one juxtaposition with respect to the camera, if the pattern ismisaligned slightly during a subsequent recognition process, the patternpose may not be precisely determinable.

As another instance, while models developed using known model trainingprocesses work well for recognizing instances of some patterns ofinterest, experience has shown that known training processes cangenerate non-robust or less than optimal models for some types ofpatterns. One pattern type that known model training processes are oftenunable to generate robust models for includes patterns that haverepeating elements such as the pattern that results when a ball gridarray is imaged. To this end, at least some existing model trainingprocesses use a training image to generate a series of images of atraining pattern at different image resolutions. The processes thencollect statistics pertaining to the spatial distribution of edges ineach image in the series known as weights. The processes then pick amaximum coarse image resolution for the model based on the circularmoment of inertia of these weights with a compensating factor for scale.Here, the compensating factor ensures that the process does not alwaysselect a finer image resolution because the edges are more disparate.These processes tend to select maximum image resolutions such that thetrained models contain relatively straight or slowly curving long chainsof strong edges that are at a distance from the model center.

When one of the training processes described above is used to developmodels and a model sequence to be used during a subsequent patternrecognition process where the pattern to be recognized includesrepeating elements, the resulting recognition process often results inidentification of an incorrect pose (i.e., a pose where the detectedpattern instance is misaligned with the actual pattern instance in theimage).

Thus, it would be advantageous to have a better recognition systemtraining process for selecting an optimal initial model including anoptimal initial resolution for use during subsequent recognitionprocesses.

BRIEF SUMMARY OF THE INVENTION

At least some embodiments of the present invention include a method forselecting a coarse image resolution or resolutions at which probe modelsmay be trained from an image of a pattern. The model(s) can then be usedat run-time where the models will be applied to different images at theselected resolutions to determine the approximate poses of occurrencesof the training pattern in those images. To this end, lower resolutionimages will be generated from both the training image and all run-timeimages using a methodology that is well known in the literature andrequires that all or much of the energy in the original image above theNyquist frequency be removed by applying an anti-aliasing filter priorto actual sub-sampling.

In other embodiments the lower-resolution images might be generated frommultiple full resolution images.

A criterion for selecting the coarse resolution or resolutions is thatthe pattern can be located with sufficient accuracy at the coarseresolution or resolutions to allow subsequent refinement steps of thealgorithm to take these hypothetical occurrences and their associatedposes and discover the exact poses of all instances of the trainingpattern within the images. Two further criteria are that the processhypothesizing occurrences and their approximate poses will not miss anyreal instances of the pattern and that the refinement step will not beoverwhelmed with so many coarse resolution hypotheses as to cause thepattern location process to take a prohibitively long time.

To this end, in at least some embodiments the invention trains models ateach of the possible coarse resolutions and tests these models using thetraining image to ascertain whether the models meet the prescribedcriteria. In addition, the invention selects a coarse resolution thatboth ensures the necessary accuracy and minimizes the amount of run-timecomputation.

In some embodiments the process of finding a pattern location using atrained probe based model entails applying the probes at a plurality ofposes (which for example may vary in rotation, scale and aspect) at eachpossible location in the image. The result of this process is acollection of score-spaces in which peaks indicate a high correspondencebetween the image and model both at that particular xy location and theparticular rotation, scale and aspect ratio at which the probes wereapplied.

Typically there is a one-to-one correspondence between the number ofpixels in an image and score-space entries. However it had beenrecognized in previous probe based pattern location algorithms that itmay not be necessary to evaluate the probe model at every possible pixellocation in the image-space and thus the score-space may be, in thissense, under-sampled. In U.S. Pat. No. 7,016,539 the degree ofunder-sampling was defined by the “scan code”, i.e. the scanning patternover which the score-space was evaluated, where a full scan code impliesgenerating scores at every pixel location, a half scan code impliesevaluating scores at every other pixel location and a quarter scan codeat only one out of every four pixels. Other scan codes are alsocontemplated (e.g. ninth scan code, etc.).

Under-sampling the score-space is different than examining a lowerequivalent resolution image with a lower equivalent resolution probemodel. Image resolution affects the appearance and existence of imagefeatures whereas this under-sampling the score space just means theprobe model will not be evaluated at every pixel location in the image.However, as is the case for choosing a coarse image resolution whereonly a subset of the possible candidate image resolutions might bedeemed suitably accurate for identifying instances of a particularpattern, a particular scan code may be required for location of anindividual pattern. Thus, in at least some embodiments it is desirablethat a training algorithm be capable of automatically identifying both arobust and a fast scan code to satisfy the constraints of the patternlocation system.

The invention described here is multifaceted in that it may simply beused to select robust coarse image resolution at which to train andstore a probe model. Another manifestation might instead select ascan-code at which to run a probe model. A further refinement mightjointly select both an image resolution and a scan code.

Thus, in embodiments where an algorithm is simply selecting the bestpossible coarse image resolution and where an exemplary range ofpossible image resolutions is from 1 to R_(max), then the selection ofthe coarsest image resolution of R_(max) would allow the algorithm torun in the fastest time. However at an image resolution of R_(max) thealgorithm may not be able to find the pattern with sufficient accuracyor may not be able to locate the pattern at all, but at a finerresolution the algorithm, though slower, might be able to find thepattern with sufficient accuracy. To ascertain which image resolution isbest, in one embodiment the invention may begin with a training imageand convert this image to a resolution of R_(max) and then train a probemodel from the image resolution of R_(max). The algorithm then tests therobustness (where robustness may be defined as locating the pattern inall test cases with sufficient accuracy) of this model and if deemedsufficiently robust will identify this image resolution as the coarsestpossible image resolution and store this model and image resolution foruse at run-time. If the resolution was found to not be robust, theoriginal training image would be converted to a finer image resolution,for example at a resolution of R_(max)−1, and a model would be trainedfrom the image resolution of R_(max)−1. The model corresponding tocoarse resolution R_(max)−1 would then be tested and the iterativeprocess of testing candidate resolutions would continue until arrivingat a training pattern which is locatable with sufficient robustness, andthis image resolution and the associated model and image resolutionwould be stored for later use.

In a further instance, where the range of possible image resolutions isfrom 1 to R_(max) and scan codes are supported by a recognition systemand include full, half and quarter, an exemplary fastest resolution/scancode combination may include an image resolution of R₁ and the quarterscan code meaning that during a pattern recognition process, after aninitial image including a pattern is obtained, the image is converted toan image having an image resolution of R₁, a model associated with animage resolution of R₁ is used to generate a score space for the imageand the score space is sampled using the quarter scan code to produce aninitial pattern location estimate. Thereafter, the initial image may beconverted to an image having an image resolution of R₂ where R₂<R₁, amodel associated with an image resolution of R₂ may be used to generatea score space for the image and the score space may be sampled using afastest robust scan code associated with the image resolution of R₂ tomore precisely identify the pattern location. This iterative processcontinues until the precise pattern location is identified using thefinest image resolution and associated model and scan code.

It has also been recognized that, in some embodiments, a similar processcan be performed to identify at least one robust image resolution andthe process may stop at that point. In other cases the process maycontinue until a plurality of robust resolutions or a plurality ofrobust image resolution and scan code combinations are identified andstored for subsequent use. The processes described herein may be used todevelop recognition processes for use with any pattern type and isparticularly useful for developing processes for use with patterns thathave repeating elements such as a ball grid or the like.

Image resolutions need not be restricted to integer values and theirprogression in the testing sequence need not be linear. For example itmight be useful to ensure more finer resolutions are tested and asequence such as 1, 1.25, 1.5, 1.75, 2.0, 2.33, 2.66, 3.0, 3.5, 4.0, 5,6, 7, etc. might be chosen.

In some cases finding the training pattern with sufficient accuracymight mean always finding the pattern within a translation tolerancedictated by the capabilities of further refinement steps of the patternlocation system to be able to refine the coarse hypothesis of thelocation of the pattern to a higher specified level of accuracy. Forexample, a refinement step might be specified to be able to takehypotheses that were within plus or minus 3 pixels of the correctlocation of the instance of the pattern and find the correct location ofthe pattern in the image, in which case the sufficient accuracy for thecoarse step would be plus or minus 3 pixels.

In further cases finding the training pattern with sufficient accuracymight mean always finding the pattern within spatial tolerance andwithin a tolerance in the further degrees of freedom of rotation, scale,shear and aspect, or some combination of these degrees of freedom.

In a further case finding the training pattern with sufficient accuracymight mean always finding the pattern within a translation tolerancedictated by the frequency of repeating features within the trainingpattern. For example, if the training image was a ball grid array andthe balls were spaced at a distance of 10 pixels then sufficientaccuracy might be defined to be a fraction of this distance, for exampleone half of the pitch of the ball grid array or 5 pixels.

In a still further case finding the training pattern with sufficientaccuracy might mean always finding the pattern within translationtolerance dictated by the frequency content of the training pattern. Forexample, if a strong peak in the frequency spectrum of the trainingimage occurred at 0.05 cycles per pixel (i.e. a cycle length of 20pixels) then sufficient accuracy might be defined to be a fraction ofthis cycle length, for example 0.4 times this cycle length or 8 pixels.

In a further case finding the training pattern with sufficient accuracymight be defined as finding the pattern with a tolerance comprising anycombination of the above definitions or other definitions.

In the embodiment where the sufficient accuracy included always findingthe pattern within translation tolerance dictated by the frequency ofrepeating features within the training pattern some embodiments of theinvention might discover this frequency by (a) training a probe model atthe full or finest image resolution, (b) using this model to generate ascore space, (c) identifying the peak pertaining to the true patternposition in the training image within this score-space, (d) identifyingthe closest significant other peak to the peak pertaining to the truepattern position in the training image within this score-space, and (e)measuring the distance between these two peaks as the distance betweenrepeating features within the training pattern, the reciprocal of whichwould be the repeating pattern frequency. The measure of sufficientaccuracy for verifying that a probe model is capable of locating thepattern in an image at a particular image resolution might be that itmust be able to locate the pattern within a fraction of the measuredrepeating pattern distance. In some cases the fraction is less than 60percent. In some cases the threshold percentage is less than 35 percent.

In some cases the verification of a probe model trained at particularimage resolution includes (a) generating a set of images at thatparticular lower resolution and also at different coarse subpixel phasesto form a set of test images, (b) using the trained model associatedwith the lower image resolution to locate the training pattern in eachof the test images, (c) measuring the location error for each of thetest images, and (d) observing whether all of these measured locationerrors fall within the bounds dictated by the algorithm's definition ofsufficient accuracy, which may be a percentage of the repeating patternfrequency or a different definition of sufficient accuracy.

In some other cases the verification of a probe model trained at aparticular image resolution might include generating an alternative setof test images that might include those generated at a particular set oftransforms (e.g. a set of rotations angles or a set of scale changes).Such an alternate set of test images might also be generated atdifferent coarse subpixel phases at for example each of the differentrotation angles.

In some embodiments verification of suitability of a probe model trainedat a particular image resolution might further include the steps of (a)specifying a minimum number of probes that are required for a probemodel covering a particular image area, (b) training probe models ateach candidate image resolution, and (c) determining if the model has atleast the minimum number of probes.

In some embodiments of the invention, verification of suitability of aprobe model at a particular image resolution might also include findingthe sparsest robust scan code for that model. In some cases the step ofidentifying the sparsest robust scan code for a particular imageresolution includes (a) generating from the training image an image atthe particular image resolution, (b) training a probe model at theparticular image resolution, (c) applying the probe model at theparticular resolution to the image at the particular resolution whileusing a mode of operation including the use of a full scan code togenerate a score space for the model applied to that image and (d)analyzing the obtained score space to ascertain whether one or moresparse scan codes would be robust in robustly detecting and accuratelylocating the pattern associated with the trained model.

Because a scan code other than the full scan code implies by definitiona sparse evaluation of a probe model's score space it is possible thatthe score corresponding to the true location of the pattern within theimage will not be evaluated. In this case, depending on the particularscan code employed, a combination of the scores around the true locationwould be observed by the algorithm. For example, if a half scan code isused, then every other location in the score space is evaluated and thusthe worst case score for the true location would be the highest score ofthe four nearest neighbors of the score at the true location.

Thus in some cases the step of analyzing the obtained score space toascertain whether a particular scan code would be robust in detectingand accurately locating the pattern associated with the trained modelmight comprise (a) identifying a worst case score that would be obtainedwhen using the particular scan code by the probe model when applied atthe true location of the trained pattern within the image, (b) comparingthis lowest score to a highest other score within the score spaceoutside the region pertaining to the true peak and, (c) where the worsecase peak score is greater than a threshold percentage of the highestother score, identifying the additional scan code as a possible sparsestrobust scan code for the image resolution.

In some cases the threshold percentage is greater than 100 percent.

In some cases the region surrounding the true peak outside of which themaximum other score is observed might be defined to be a 3×3 region. Inother cases the region may be defined differently.

An objective of some embodiments of the invention is to train a probemodel at an image resolution and/or scan code combination that can findwith sufficient robustness the location of the training pattern in testimages where the model facilitates the fastest possible speed ofexecution. In embodiments where only image resolution is selected, thecoarsest resolution that ensures the location of the training pattern intest images with sufficient robustness is the optimum choice. Inembodiments that only select the scan code, the most sparse scan codethat ensures the location of the training pattern in test images withsufficient robustness is the optimum choice. In some cases a higherresolution model with a sparser scan code may run faster than a coarserresolution model with a less sparse scan code. For example, a probemodel trained at an image resolution of 10 and using a half scan codemay well run slower than a probe model trained at an image resolution of8 using a quarter scan code.

In such a case the steps for finding the best combination of imageresolution and scan code might include, starting with the coarsest imageresolution as the ‘proposed’ image resolution and (a) generating a lowerresolution image at the proposed image resolution from the fullresolution training image, (b) using the image at the proposed imageresolution to train a probe model, (c) running the probe model on theimage at the proposed image resolution to generate a score space and alocation error, (d) using the location error to determine if the testimage resolution is sufficiently accurate in locating the pattern withina fraction of one pitch of the repeating elements or other suitabletolerance, (e) where the proposed image resolution is deemed to besufficiently robust, using the score space to identify a sparsest robustscan code for the proposed image resolution where the proposed imageresolution and sparsest robust scan code comprise a proposed imageresolution and scan code combination, (f) storing the proposed imageresolution and scan code combination for use in subsequent patternlocation processes as a stored image resolution/scan code combination,(g) determining if a less coarse image resolution and more sparse scancode combination could be faster than the stored image resolution/scancode combination, (h) where a less coarse image resolution and moresparse scan code combination cannot be faster than the stored imageresolution/scan code combination, ending the process and (i) where aless coarse image resolution and more sparse scan code combination couldbe faster than the stored image resolution/scan code combination,repeating steps (a) through (h) using a less coarse proposed imageresolution and only scan codes that would allow faster execution at theproposed less coarse image resolution than the stored imageresolution/scan code combination.

Some embodiments further include the step of using the full resolutiontraining image to generate a set of test images at a particular lowerresolution with different subpixel phases and wherein steps (c) through(e) are performed for each of the lower resolution images in this set.Further embodiments might include extending the set of test images toinclude those generated at various combinations of rotation, scale,shear and aspect in addition to sub pixel phase. Such an embodimentmight also include in its definition pattern location error,discrepancies in rotation, scale, shear and aspect between the trueinstance of the pattern in the test image and that determined byapplying the trained probe model to the test image.

In a further embodiment of the invention, the invention may be used togenerated a selected coarse image resolution or resolutions and/orsparse scan codes and these may then be stored for later manual entry ina run-time pattern recognition or pattern location system.

Consistent with the above comments, some embodiments of the inventioninclude a method for training a pattern recognition algorithm for amachine vision system that uses models of a pattern to be located, themethod comprising the steps of training each of a plurality of modelsusing a different training image wherein each of the training images isa version of a single image of the pattern at a unique image resolution,using the models to identify at least one robust image resolution wherethe resolution is suitable for locating the pattern within an accuracylimit of the actual location of the pattern in the image and storing theat least one robust image resolution for use in subsequent patternrecognition processes.

In some cases each model includes a plurality of probes that are each ameasure of similarity of at least one of a run-time image feature and arun time image region to at least one of a pattern feature and a patternregion, respectively, at a specific location, the method furtherincluding the steps of applying the plurality of probes at a pluralityof poses to the run-time image resulting in a plurality of score spacesat each pose where the score spaces include peaks, the peaks indicatinglikely locations of occurrences of the pattern in the run-time image ateach pose and comparing the score-space peaks to an accept thresholdwhere the results of the comparison are used to identify the poses ofinstances of the model in the image.

In some cases the pattern recognition algorithm searches for the patternin a translation degree of freedom and at least one further degree offreedom.

In some cases the step of using the models to identify at least onerobust image resolution that is suitable for locating the pattern withinan accuracy limit of the actual location of the pattern in the imageincludes using the models to identify at least one image resolutionusable to find the pattern within an accuracy limit that at least one of(i) is a function of the capabilities of further refinement steps of therecognition algorithm to be able to refine the location of the patternto a higher level of accuracy; (ii) is within a spatial, a rotation, ascale, a shear and an aspect degree of freedom; (iii) is a function ofthe frequency content of the training pattern and (iv) any combinationof (i) through (iii) above.

In some cases the step of using the models to identify at least onerobust image resolution where the robust image resolution is suitablefor locating the pattern within an accuracy limit of the actual locationof the pattern in the image includes using different models associatedwith different image resolutions to examine images until at least onerobust image resolution is identified and wherein the order of modelsused includes one of (i) starting with a coarse resolution model andlinearly progressing to finer resolution models via integers; (ii)starting with one resolution and progressing in a non-linear fashion toother resolutions; and (iii) using at least some non-integerresolutions.

In some cases the step of using the models to identify at least onerobust image resolution where the robust image resolution is suitablefor locating the pattern within an accuracy limit of the actual locationof the pattern in the image includes generating a plurality of testimages at the robust image resolution, each test image characterized bya different combination of degrees of freedom where the degrees offreedom include at least a sub-set of rotation, shear, aspect and scalevalues and applying the model to each of the plurality of test images todetermine if pattern location is identified within the accuracy limit ofthe actual location of the pattern in the images.

In some cases the method further includes the step of providing therobust image resolution for manual entry into a run-time patternrecognition system. In some cases the method further includes the stepof providing the robust image resolution/scan code combination formanual entry into a run-time pattern recognition system.

Some embodiments further include the step of identifying at least oneimage resolution and scan code combination where the image resolution issuitable for locating the pattern within the accuracy limit of theactual location of the pattern in the image and the scan code is robustat the image resolution in the image resolution/scan code combination,the step of storing the at least one robust resolution including storingat least one robust image resolution and scan code combination. In somecases the pattern includes repeating elements and wherein the step ofusing the models to identify at least one robust image resolutionincludes using the models to identify at least one image resolutionwhere the image resolution is suitable for locating the pattern within afraction of one pitch of the repeating elements.

Some embodiments include an apparatus for training a pattern recognitionalgorithm for a machine vision system that uses models of a pattern tobe located, the apparatus comprising a processor programmed to performthe steps of, training each of a plurality of models using a differenttraining image wherein each of the training images is a version of asingle image of the pattern at a unique image resolution, using themodels to identify at least one robust image resolution where the imageresolution is suitable for locating the pattern within an accuracy limitof the actual location of the pattern in the image and storing the atleast one robust image resolution for use in subsequent patternrecognition processes. In some cases each model includes probes thateach represent a relative position at which a test is to be performed inan image at a given pose where the algorithm compares the models withrun time images at each of a plurality of poses and computes a matchscore at each of the poses thereby providing a match score space whichis compared to an accept threshold where the results of the comparisonare used to identify the poses of instances of the model in the image.

In some cases the processor is further programmed to perform the step ofidentifying a fastest image resolution and scan code combination wherethe image resolution is suitable for locating the pattern within theaccuracy limit of the actual location of the pattern in the image andthe scan code is robust at the image resolution in the imageresolution/scan code combination, the processor programmed to performthe step of storing the at least one robust image resolution by storingat least one robust image resolution and scan code combination. In somecases the pattern includes repeating elements and wherein the processoris programmed to perform the step of using the models to identify atleast one robust image resolution by using the models to identify atleast one image resolution where the image resolution is suitable forlocating the pattern within a fraction of one pitch of the repeatingelements.

In some cases the processor is further programmed to perform the step ofidentifying the known location of the pattern within an image andwherein the processor is programmed to perform the step of using themodels to identify the fastest combination by using a first modelassociated with a finest image resolution to generate a first scorespace for a first image, examining the first score space to identify arepeating pattern frequency, for each of the other models associatedwith different resolutions, using the model to generate an additionalscore space and a location error, the location error indicating thedifference between the known location and a location determined usingthe additional score space, where the location error is less than athreshold percentage of the repeating pattern frequency, identifying theimage resolution as a possible image resolution for the fastest imageresolution and scan code combination.

In some cases the processor is programmed to perform the step of usingthe model to generate an additional score space by generating a set oflower resolution images at different coarse subpixel phases and usingthe model associated with the image resolution to generate a separateadditional score space and a location error for each of the lowerresolution images, the processor programmed to perform the step ofidentifying the image resolution as a possible image resolution for thefastest image resolution and scan code combination by identifying theimage resolution as a possible image resolution for the fastest imageresolution and scan code combination when each of the location errorsgenerated using the model is less than a threshold percentage of therepeating pattern frequency. In some cases the processor is programmedto perform the step of using the models to identify the fastest imageresolution and scan code combination by, for each of the possibleresolutions, identifying the sparsest robust scan code.

In some cases the processor is programmed to perform the step ofidentifying the sparsest robust scan code for a particular imageresolution by examining the additional score space generated using themodel associated with the image resolution using at least one additionalscan code that is more sparse than the full scan code to determine ifthe additional scan code is robust at the particular image resolution.In some cases the processor is programmed to perform the step of usingthe additional scan code to examine the score space by identifying apeak score in the score space, identifying a worse case peak score in ascan region centered on the peak score in the score space, comparing theworst case peak score to a highest other score within the score spaceoutside the scan region and, where the worse case peak score is greaterthan a threshold percentage of the highest other score, identifying theadditional scan code as a possible sparsest robust scan code for theimage resolution. In some cases the additional scan code includes atleast one of a quarter scan code and a half scan code.

In some cases the processor is programmed to perform the steps oftraining, using and storing by, starting with a coarse image having acoarse image resolution where the coarse image resolution is a testimage resolution (a) using the image at the test image resolution totrain a model, (b) running the model on the image at the test imageresolution to generate a score space and a location error, (c) using thelocation error to determine if the test image resolution is suitable forlocating the pattern within a fraction of one pitch of the repeatingelements, (d) where the test image resolution is suitable for locatingthe pattern within a fraction of one pitch of the repeating elements,using the score space to identify a sparsest robust scan code for thetest image resolution where the test image resolution and sparsestrobust scan code comprise a test image resolution and scan codecombination, (e) storing the test image resolution and scan codecombination for use in subsequent pattern recognition processes as astored image resolution/scan code combination, (f) determining if a lesscoarse image resolution and scan code combination could be faster thanthe stored image resolution/scan code combination, (g) where a lesscoarse image resolution and scan code combination cannot be faster thanthe stored image resolution/scan code combination, ending the processand (h) where a less coarse image resolution and scan code combinationcould be faster than the stored image resolution/scan code combination,repeating steps (a) through (g) using a less coarse test imageresolution and only scan codes that could be faster at the less coarsetest image resolution than the stored image resolution/scan codecombination.

In some cases the processor is further programmed to perform the stepsof identifying the known location of the pattern within a finest imageresolution image, using an image at the finest image resolution to traina model, using the model associated with a finest image resolution togenerate a score space and examining the score space to identify arepeating pattern frequency, the processor programmed to perform thestep of using the location error to determine if the test imageresolution is suitable for locating the pattern within a fraction of onepitch of the repeating elements by comparing the location error to athreshold percentage of the repeating pattern frequency and when thelocation error is less than the threshold percentage of the repeatingpattern frequency, determining that the test image resolution issuitable for locating the pattern within a fraction of one pitch of therepeating elements.

In some cases the processor is programmed to perform the step of usingthe models to identify at least one robust image resolution that issuitable for locating the pattern within an accuracy limit of the actuallocation of the pattern in the image by using the models to identify atleast one image resolution usable to find the pattern within an accuracylimit that at least one of (i) is a function of the capabilities offurther refinement steps of the recognition algorithm to be able torefine the location of the pattern to a higher level of accuracy; (ii)is within a spatial, a rotation, a scale, a shear and an aspect degreeof freedom; (iii) is a function of the frequency content of the trainingpattern and (iv) any combination of (i) through (iii) above.

In some cases the processor performs the step of using the models toidentify at least one robust image resolution where the robust imageresolution is suitable for locating the pattern within an accuracy limitof the actual location of the pattern in the image by using differentmodels associated with different image resolutions to examine imagesuntil at least one robust image resolution is identified and wherein theorder of models used includes one of (i) starting with a coarseresolution model and linearly progressing to finer resolution models viaintegers; (ii) starting with one resolution and progressing in anon-linear fashion to other resolutions; and (iii) using at least somenon-integer resolutions.

In some cases the processor performs the step of using the models toidentify at least one robust image resolution where the robust imageresolution is suitable for locating the pattern within an accuracy limitof the actual location of the pattern in the image by generating aplurality of test images at the robust image resolution, each test imagecharacterized by a different combination of degrees of freedom where thedegrees of freedom include at least a sub-set of rotation, shear, aspectand scale values and applying the model to each of the plurality of testimages to determine if pattern location is identified within theaccuracy limit of the actual location of the pattern in the images.

In some cases the processor is further programmed to provide the robustimage resolution for manual entry into a run-time pattern recognitionsystem. In some cases the processor is further programmed to providingthe robust image resolution/scan code combination for manual entry intoa run-time pattern recognition system.

Still other embodiments include an apparatus for training a patternrecognition algorithm for a machine vision system that uses models of apattern to be located, the apparatus comprising a processor programmedto perform the steps of (a) identifying the known location of thepattern that includes repeating elements within a finest imageresolution image, (b) using the finest image resolution image to train amodel, (c) using the model to examine the finest image resolution imageand to generate a score space, (d) examining the score space to identifya repeating pattern frequency, (e) using a coarse image at a coarseimage resolution that is coarser than the finest image resolution imageto train a model, (f) using the model associated with the coarse imageto examine the coarse image thereby generating a location error, (g)comparing the location error to the repeating pattern frequency, and (h)based on the comparison, determining if the coarse image resolution issuitable for locating the pattern within a fraction of one pitch of therepeating elements.

In some cases the processor is programmed to further perform the stepsof using the coarse image to generate a plurality of lower resolutionimages at different coarse subpixel phases and wherein steps (f) through(h) are performed for each of the lower resolution images.

Other embodiments include an apparatus for training a patternrecognition algorithm for a machine vision system that uses models of apattern to be located, the apparatus comprising a processor programmedto perform the steps of (a) identifying the known location of thepattern that includes repeating elements within a finest imageresolution image, (b) using the finest image resolution image to train amodel, (c) using the model to examine the finest image resolution imageand to generate a score space, (d) using a coarse image at a coarseimage resolution that is coarser than the finest image resolution imageto train a model, (e) using the model associated with the coarse imageto examine the coarse image thereby generating a score space and (f)examining the score space to identify a sparsest scan code that isrobust at the coarse image resolution.

In some cases the processor is programmed to further perform the stepsof using the coarse image to generate a plurality of lower resolutionimages at different coarse subpixel phases and wherein steps (e) and (f)are performed for each of the lower resolution images.

Other embodiments include a method for training a pattern recognitionalgorithm for a machine vision system that uses models of a pattern tobe located, the method comprising the steps of generating a test imageat a fine image resolution, using the test image to generate a pluralityof images where each of the plurality of images is at a different testimage resolution, generating lower resolution images for each of theplurality of images, at each of a plurality of the test imageresolutions, using a model to examine the lower resolution images toidentify location errors and selecting at least one test imageresolution for which location is resolved within an accuracy limit ofthe actual location of the pattern within the test image.

In some cases the step of selecting at least one image resolutionincludes selecting a sub-set that includes at least two testresolutions. In some cases the step of selecting at least one imageresolution includes selecting at least the fastest image resolution forwhich location is resolved within the accuracy limit.

To the accomplishment of the foregoing and related ends, the invention,then, comprises the features hereinafter fully described. The followingdescription and the annexed drawings set forth in detail certainillustrative aspects of the invention. However, these aspects areindicative of but a few of the various ways in which the principles ofthe invention can be employed. Other aspects, advantages and novelfeatures of the invention will become apparent from the followingdetailed description of the invention when considered in conjunctionwith the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic illustrating an exemplary machine vision systemincluding a computer that may run a pattern recognition training programthat is consistent with at least some aspects of the present invention;

FIG. 2 is a top plan view of a repeating element ball grid pattern;

FIG. 3 is a flow chart illustrating an exemplary method that isconsistent with the present disclosure for identifying a fastest imageresolution and scan code combination for use in subsequent patternrecognition processes;

FIG. 4 is a schematic illustrating an exemplary score space that may begenerated during a portion of the process shown in FIG. 3;

FIG. 5 is a schematic illustrating a sub-sampling process where lowerresolution images are generated at a different coarse subpixel phases;

FIG. 6 is a sub-process that may be substituted for a portion of theprocess shown in FIG. 3 for identifying the sparsest robust scan codefor a test image resolution;

FIG. 7 is a schematic illustrating an exemplary region of a score spaceto be analyzed during full and half-scan examination processes in FIG.6;

FIG. 8 is a flow chart illustrating another exemplary process that isconsistent with at least some aspects of the present invention;

FIG. 9 is a flow chart illustrating a sub-process that may besubstituted for a portion of the process shown in FIG. 3;

FIG. 10 is a flow chart illustrating a sub-process that may besubstituted for a portion of the process shown in FIG. 3;

FIG. 11 is a flow chart illustrating a sub-process that may besubstituted for a portion of the process shown in FIG. 3; and

FIG. 12 is a flow chart illustrating a general process whereby a robusttest image resolution is selected for use as an initial image resolutionduring subsequent recognition processes.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof have been shown by wayof example in the drawings and are herein described in detail. It shouldbe understood, however, that the description herein of specificembodiments is not intended to limit the invention to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE INVENTION

The various aspects of the subject invention are now described withreference to the related figures, wherein like numerals refer to like orcorresponding elements throughout. It should be understood, however,that the drawings and detailed description relating thereto are notintended to limit the claimed subject matter to the particular formdisclosed. Rather, the intention is to cover all modifications,equivalents, and alternatives falling within the spirit and scope of theclaimed subject matter.

As used herein, the terms “component,” “system” and the like areintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on computer and the computercan be a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs.

Furthermore, the disclosed subject matter may be implemented as asystem, method, apparatus, or article of manufacture using standardprogramming and/or engineering techniques to produce software, firmware,hardware, or any combination thereof to control a computer or processorbased device to implement aspects detailed herein. The term “article ofmanufacture” (or alternatively, “computer program product”) as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips . . . ), opticaldisks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ),smart cards, and flash memory devices (e.g., card, stick). Additionallyit should be appreciated that a carrier wave can be employed to carrycomputer-readable electronic data such as those used in transmitting andreceiving electronic mail or in accessing a network such as the Internetor a local area network (LAN). Of course, those skilled in the art willrecognize many modifications may be made to this configuration withoutdeparting from the scope or spirit of the claimed subject matter.

Referring now to the drawings wherein like reference numerals correspondto similar elements throughout the several views and, more specifically,referring to FIG. 1, the present invention will be described in thecontext of an exemplary machine vision system 10 including a camera 12,a computer 14 that includes a processor and a human-machine interface 20in the form of a computer display and optionally one or more inputdevices (e.g., a keyboard, a mouse, a track ball, etc.). Camera 12includes, among other components, a lens 24 and a camera sensor element(not illustrated). Lens 24 includes a field of view 26 and focuses lightfrom the field of view 26 onto the sensor element. The sensor elementgenerates a digital image of the camera field of view and provides thatimage to a processor that forms part of computer 14. In at least someembodiments the camera sensor includes a two-dimensional CCD or CMOSimaging array as well known in the art.

Server 14 may be linked to one or more databases (not illustrated) thatstore programs run by server 14 and databases used by the programs runby the server. In addition to other programs, server 14 runs a patternrecognition training program that can be performed to, as the labelimplies, develop a pattern recognition process for locating a specificpattern within images generated by camera 12. More specifically, whileserver 14 may be able to run various pattern recognition trainingprograms for generally recognizing patterns within images, consistentwith various aspects of the present disclosure, server 14 can run aspecial recognition training program useable to develop a robust patternrecognition process for locating instances of patterns that includerepeating elements within images. Hereafter, unless indicated otherwise,server 14 will be referred to as a processor 14.

While the pattern recognition training programs or processes describedherein may be used to develop processes particularly suitable foridentifying instances of any patterns that have repeating elements, inorder to simplify this explanation, the present invention will bedescribed in the context of an exemplary repeating element ball gridpattern. To this end, referring now to FIG. 2, an image of an exemplaryball grid array 28 is illustrated that, as the label implies includesballs of solder 30 that are arranged in a grid including equi-spacedrows and columns. Clearly array 28 shown in FIG. 2 includes repeatingelements within a pattern.

Referring once again to FIG. 1, interface 20 is linked to server 14 sothat a system operator can, in at least some cases, interact with thepattern recognition training program to help develop the trainingprogram. To this end, for instance, when server 14 receives an initialtraining image of array 28 from camera 12, that initial image includingthe array pattern may be presented via interface 20 to the operator.Here, the operator may use one of the input devices to identify thepattern to be recognized during subsequent pattern recognitionprocesses. Once a pattern is identified, the location of the patternwithin the image generated by camera 12 is identified or is preciselyknown by the server 14 and can be stored for subsequent purposes.

In general, at least one pattern recognition training programcontemplated by this disclosure uses a fine image resolution imagegenerated by camera 12 to generate other coarser resolution images.Thereafter, the training program performs processes to determine whichcoarse images can be used to identify pattern locations accuratelyenough that subsequent finer image resolution images will be able toprecisely identify the pattern location. In addition, where a coarseimage resolution image can be used to identify pattern locationsaccurately enough, the program identifies a sparsest (i.e., fastest)robust scan code that can be used at the image resolution for generatinga location estimate. The training program selects the fastest imageresolution and scan code combination that is sufficiently robust andstores that combination as a starting point for subsequent patternrecognition processes.

U.S. Pat. No. 7,016,539 which is incorporated above by reference teachesan exemplary probe model algorithm or process that uses models of apattern of interest to be found where each model includes a plurality ofprobes and each probe represents a relative position at which a test isto be performed in an image at a given pose. Each test contributesevidence that a pattern exists at the pose. The models are used toexamine a run time image at each of a plurality of poses and to computea match score at each of the poses thereby providing a match scorespace. The match scores are compared to an accept threshold and theresults of the comparison are used to provide the locations of anyinstances of the model in the image. This patent should be referred tofor details regarding models and how those models are used during arecognition process.

Referring now to FIG. 3, an exemplary pattern recognition trainingmethod 50 is illustrated that is consistent with at least some aspectsof the present disclosure. Referring also to FIG. 1, at block 52 asystem operator uses terminal 20 to specify a minimum number of probesthat must be included in each model to be developed during the trainingprogram. Here, experience has shown that where a model includes too fewprobes, the model does not work properly in some cases. As a default,the minimum number of probes may be set to 64 but, in any event, shouldnot be less than the number of pixels in an image (i.e., the width inpixels times the height in pixels) times 64 divided by 10,000 in atleast some embodiment. At block 53, camera 12 is used to generate a testimage of a ball grid array 28 where the test image has the finestpossible image resolution. At block 54, server 14 trains or generates amodel for the finest image resolution test image in a manner consistentwith that described in U.S. Pat. No. 7,016,539 which is incorporatedabove.

Referring still to FIGS. 1 and 3, at block 56, processor 14 uses thetrained model to examine a full image resolution image in full scan modeto generate a score space as described in U.S. Pat. No. 7,016,539.Referring to FIG. 4, an exemplary score space 122 is illustrated wherelocal peak values are identified by numeral 124. At block 58, arepeating pattern frequency (RPF) between a highest local peak score anda nearest peak in score space 122 is identified. Here, the repeatingpattern frequency is identified by a possibly non-integer number ofpixels between the highest and nearest peak scores. For example, anexemplary RPF may be 19 pixel within the score space 122.

Referring still to FIGS. 1 and 3, at block 60, a test image resolution(TR) is set equal to a maximum test image resolution that is supportedby the processor 14. Here, it is assumed that processor 14 supports arange of image resolution between 1 and 25 and therefore, the maximum TRis set equal to 25 at block 60. In addition, a flag is set equal to zeroat block 60. The flag in block 60 is used to store a first sufficientlyaccurate test image resolution and full scan code combination duringsubsequent steps in the process 50 as will be described in greaterdetail below.

In FIG. 3, at block 62, processor 14 uses the test image from block 53to generate a coarser test image at the current test image resolution(e.g., at a image resolution of 25 initially) and to build or train amodel using that coarser test image. To this end, referring also to FIG.5, a pixel raster 114 corresponding to a portion of an image isillustrated where the raster includes separate pixels, three of whichare collectively identified by numeral 118. In FIG. 5, each of the smallsquares represents a separate pixel 118. To combine the pixels in animage to generate an image having a coarser image resolution, groups ofadjacent pixels are combined. For example, to generate an image having atest image resolution of 4, the 16 pixels in box 116 (i.e., a 4×4region) are combined to form a single pixel. Similarly, the 16 pixels inbox 117 are combined into a single pixel, the 16 pixels in box 119 arecombined into a single pixel and the 16 pixels in box 121 are combinedinto a single pixel. Where the test image resolution is 10 as opposed to4, 100 pixel regions would be combined into a single pixel, and so on.

Referring still to FIG. 3, after a model has been trained for the testimage resolution at block 62, control passes to block 64. At block 64,processor 14 determines whether or not the trained model includes aminimum number of probes (e.g., 64 in at least some embodiments). Wherethe trained model does not include a minimum number of probes, controlpasses to block 65 where the value of the flag set at block 60 iscompared to zero. Here, where the flag is zero the process has yet toverify an image resolution/scan code combination that will work as astarting point for subsequent pattern recognition processes andtherefore the process must continue and control passes to block 72 wherethe test image resolution TR is decremented by one or reduced by somesuitable value. After block 72, the adjusted test image resolution TR iscompared to one at block 73. Where the test image resolution is lessthan the original fine resolution which in most cases is one, theprocess ends. Where the test image resolution is greater than or equalto one, control passes back up to block 62 where the process continuesas described above.

Referring still to FIG. 3, at block 65, where the flag is not zero(i.e., the flag is one), control passes to block 86 where processor 14determines if the next test image resolution is possibly faster than thestored and verified image resolution/scan code combination. Where thenext possible test image resolution could not possibly be faster thanthe stored/verified image resolution/code combination, the process ends.Where the next possible test image resolution could possibly be fasterthan the stored/verified image resolution/code combination, controlpasses on to blocks 72 and 73 described above. The loop including blocks64, 65, 86, 72 and 73 continues until a model is trained at block 62that includes the minimum number of probes.

Once a model includes a minimum number of probes at block 64, controlpasses to block 66 where processor 14 uses the finest resolution testimage from block 53 to generate lower resolution images at differentcoarse subpixel phases at the current test image resolution. To thisend, referring again to FIG. 5, where regions 116, 117, 119 and 212correspond to a (0.0, 0.0) phase coarse image resolution image at a testimage resolution of 4, the (0.50, 0.50) phase image at a test imageresolution of 4 is phase shifted by one-half the test image resolutionpixels along x and y axes and therefore would include 4×4 pixelscorresponding to regions 123, 125, 127 and 129, among other regions.While various numbers of lower resolution images may be generated atblock 66, in at least some embodiments the lower resolution images wouldinclude phases (0.25, 0.25), (0.25, 0.75), (0.75, 0.25), (0.75, 0.75)and (0.50, 0.50).

Referring still to FIGS. 1 and 3, at block 68 the model trained at block62 for the current test image resolution is used to examine each of thelower resolution images at full scan code to generate location errors inthe image space and a separate score space for each one of the lowerresolution images. The location errors here would be specified as apossibly non-integer number of pixels. At block 70, processor 14compares the location errors from block 68 to the repeating patternfrequency that was determined at block 58. Where each of the locationerrors is less than a threshold percentage of the repeating patternfrequency (i.e., less than a desired accuracy distance or accuracylimit), control passes up to block 74. Where any one of the locationerrors is greater than a threshold percentage of the repeating patternfrequency (i.e., is greater than a desired accuracy distance) at block70, control passes to block 65 where a subset of the sequence ofsub-processes including blocks 65, 86, 72 and 73 occur and control maypass back up to block 62. At block 70, the threshold percentage isselected to minimize or eliminate conditions in which the patternlocation errors occur. In some embodiments, the threshold percentage maybe as high as 60%. In particularly advantageous embodiments, it has beendetermined that the threshold percentage should be approximately 35%because, when the threshold is 35%, the test image resolution andrelated model are suitable for locating the repeating pattern within afraction of one pitch of the repeating elements.

Here it should be appreciated that while one accuracy limit isrepresented by the expression in block 70, many other accuracy limits orerror thresholds are contemplated and the inventions should not belimited to the specific expression unless so claimed. For instance, insome embodiments finding a pattern with sufficient accuracy may meanalways finding a pattern within an accuracy limit dictated by thecapabilities of further refinement steps of the pattern location systemto be able to refine the coarse pattern location hypothesis to a higherlevel of accuracy. In other cases finding a pattern with sufficientaccuracy might mean finding the pattern within a tolerance in anysub-set of various degrees of freedom including rotation, scale, shearand aspect. In still other cases finding a pattern with sufficientaccuracy may mean finding the pattern within an accuracy limit dictatedby the frequency content of the training pattern. In some cases acombination of the above definitions may be used to determine ifsufficient accuracy occurs.

Referring still to FIGS. 1 and 3, at block 74, where the flag set atblock 60 is still equal to zero, (i.e., the first time through block 74during a pattern recognition training process), control passes to block76 where the flag is reset to equal 1. Thus, the second time throughblock 74 during a training process, the flag will not equal zero andcontrol will skip from block 74 down to block 80. At block 78, the testimage resolution and full scan code are stored as a verified imageresolution/code meaning that the current test image resolution and thefull scan code can be used as a starting point for successfullyidentifying a repeating elements pattern in a subsequent patternrecognition process. As explained hereafter, the verified or storedimage resolution/code stored at block 78 will be replaced by other imageresolution/code combinations if other robust and faster combinations areidentified during the remainder of the training process.

Continuing, at block 80, processor 14 identifies the sparsest robustscan code at the test image resolution. At block 82, processor 14determines whether or not the test image resolution and the sparsestrobust scan code at the test image resolution are faster than theverified or stored image resolution/code combination. Here, forinstance, where the stored verified image resolution and scan data codecombination includes an image resolution of 20 and a quarter-scan code,no image resolution/scan code combination having an image resolutionless than 20 can be faster than the verified combination (if thesparsest scan code tested is quarter-scan). However, where the verifiedcombination includes an image resolution of 20 and the full-scan code,at least some combinations including resolutions that are finer than 20(i.e., less than 20) may be faster than the verified combination. Forinstance, a combination including an image resolution of 19 and eitherthe half or quarter-scan code would be faster than the 20 imageresolution/full-scan combination. As another instance, where the storedverified combination includes a test image resolution of 20 and thehalf-scan code, while a combination including image resolution of 19 andthe full-scan code might be slower, a combination of 19 and thequarter-scan code might be faster. Where the current image resolutionand sparsest robust scan code are not faster than the verified imageresolution/code, control passes back to block 86 where, as describedabove, processor 14 determines if the next test image resolution couldpossibly be faster than the verified image resolution/code combination.Where the next image resolution cannot possibly be faster, the processends. Where the next image resolution could possibly be faster than theverified image resolution/code, control passes to block 72 where thetest image resolution is decreased by 1 after which control again passesto block 73. In other embodiments the image resolution may bedecremented by 2 or more or by a non-integer value (e.g., a fraction).

Referring again to decision block 82, when the test image resolution andthe sparsest robust scan code at the test image resolution are fasterthan the verified image resolution/code, control passes to block 84where the verified image resolution/code is replaced by the test imageresolution and fastest scan code associated therewith. After block 84control passes to block 86.

Referring now to FIG. 6, a sub-process 88 for identifying a sparsestrobust scan code for a specific test image resolution and that may besubstituted for blocks 80,82, 84 and 86 in FIG. 3 is illustrated.Referring also to FIGS. 1 and 3, after the control loop including block74, 76 and 78, control passes to block 90 in FIG. 6. At block 90, foreach one of the score spaces (see again FIG. 4) generated at block 68for the lower resolution images generated at block 66, processor 14identifies a peak score location. For example, referring to FIG. 7, aregion from a score space where each location in the region includes aspecific score is shown. For the purposes of the this explanation, itwill be assumed that the score 99 in the center of 3×3 region 200 is thepeak score in the entire score space where the score space would includea much larger region (see again exemplary larger score space 122 in FIG.4) than the region shown in FIG. 7. Thus, at block 90 in FIG. 6, thelocation corresponding to score 99 would be identified. At process block92, processor 14 identifies the 3×3 region 200 about the peak scorelocation in the score space and identifies the worse case peak score fora quarter scan code. Referring again to FIG. 7, the 3×3 region examinedat block 92 includes region 200 as illustrated. Here, if a quarter scancode were run on the region 200, every fourth location in the 3×3 regionwould be evaluated. Depending on where the 3×3 region fell within thecoarse resolution image, a quarter scan evaluation might sample andevaluate score 99 or scores 62 and 48 or scores 79, 56 and 87 or scores39, 69 and 68. Thus, with a quarter scan code the worst case peak score(i.e., the peak score in any one of the four sampling groups that ispossible) would be 62.

Referring still to FIGS. 1 and 6, at block 94, the worst case peak scoreis compared to a threshold percentage of the highest other score withinthe score space outside the 3×3 region 200. Thus, in the example shownin FIG. 7, the worst case peak score 62 would be compared to a thresholdpercentage of the highest other score within the score space outsideregion 200. The threshold percentage in block 94 is selected so as tosubstantially ensure that the quarter scan code will be robust. In atleast some embodiment, as shown in block 94, the threshold percentage isset to 105%. Thus, in the present example where the worst case peakscore of 62 is greater than 105% of the highest other score within thescore space outside region 200 and where similar conditions exist foreach of the lower resolution images, control passes to block 96. Atblock 96, the verified image resolution/code is replaced with thecurrent test image resolution and quarter scan code after which theprocess ends. Thus, any time a quarter scan code is verified as robustat block 94, that quarter scan code and its associated test imageresolution will be the fastest image resolution/code combination thatcan be reliably used to start a pattern recognition process and thetraining process ends.

Referring still to FIGS. 1 and 6, at block 94 where the worst case peakscore is less than 105% of the highest other score within the scorespace outside region 200, control passes to block 98. At block 98,processor 14 determines whether or not the verified code (i.e., thestored code which forms part of the verified image resolution/codecombination) is the half-scan code. At block 98, because the processmade it past block 94, it is known that the quarter scan code is notrobust for the current test image resolution. It is also known that if ahalf-scan code was found robust for a coarser image resolution than thecurrent test image resolution, the half-scan code should not be testedfor the current test code image resolution because the combination willnot be faster than the verified image resolution/code combination.Furthermore, the full scan code is slower than the half-scan code andtherefore, if a half-scan code has been verified for a coarser imageresolution than the current test image resolution, the full scan codeshould not be examined for the current test image resolution. For thisreason, when the verified code is the half-scan code at block 98,control passes to block 110 where the test image resolution is decreasedby 1 (or some other suitable value).

At block 112, processor 14 determines whether or not the current testimage resolution is greater than the verified image resolution dividedby √{square root over (2)}. Here, we know that a quarter-scan code istwo times faster than a half-scan code and four times faster than a fullscan code. In addition, we know that a half-scan code is two timesfaster than a full scan code. Moreover, we know that an image resolutionof two is four times faster than an image resolution of one andgenerally that an image resolution of g is g² faster than an imageresolution of one. Thus, based on these known relationships, we knowthat an image resolution must be at least √{square root over (2)} timeslarger (coarser) for a half-scan to be faster than a quarter-scan. Thus,where the verified code is half-scan to continue on and to test the nextfinest test image resolution (i.e., the image resolution set at block110) to see if a quarter-scan code is robust, the next test imageresolution has to be greater than the verified image resolution (i.e.,the stored image resolution that is associated with the verified orstored half-scan code) divided by √{square root over (2)}. Where thenext test image resolution is greater than the verified image resolutiondivided by √{square root over (2)} at block 112, control passes back upto block 62 in FIG. 3 where the process described above is repeated.Where the test image resolution is not greater than the verified imageresolution divided by √{square root over (2)} at block 112, thecurrently stored and verified image resolution and half-scan coderepresent the fastest image resolution/code combination that will berobust and accurate as a starting point for subsequent positionrecognition processes and the training process ends.

Referring once again to FIGS. 1 and 6, at block 98, if the verified codeis not a half-scan code that means that the verified code is a full-scancode and therefore control passes to block 100. At block 100, processor14 determines whether or not the test image resolution is greater thanthe stored or verified image resolution divided by √{square root over(2)}. Here, an image resolution must be at least √{square root over (2)}times larger or coarser for a full-scan to be faster than a half-scanand therefore, because the verified code at 100 is a full scan,half-scan testing should not occur unless the test image resolution isgreater than the verified image resolution divided by √{square root over(2)}.

Where the test image resolution is greater than the verified imageresolution divided by the √{square root over (2)}, control passes toblock 106 where half-scan sampling is examined to determine whether ornot the half-scan code is robust at the test image resolution. To thisend, at block 106, for each of the lower resolution images generated atblock 68 (see again FIG. 3), processor 14 identifies a worst case peakscore within a 1×3 region 201 about the peak score location that wasidentified at block 90. Referring again to FIG. 7 where the peak scorelocation is at score 99, running a half scan, every other location inthe region 201 is sampled and evaluated. Depending on where the 1×3region 200 falls within a coarse image resolution image, in theillustrated example the evaluated scores may include 62 and 48 or 99.Thus, here, the worst case peak score using the half-scan code would be62.

Referring still to FIG. 6, at block 108, processor 14 compares the worstcase peak score to a threshold percentage of the highest other scorewithin the score space outside the 1×3 region 201. Once again, as inblock 94, the threshold percentage in block 108 is 105%. Where the worstcase peak score is greater than 105% of the highest other score withinthe score space outside the region 201 for each of the lower resolutionimages, control passes to block 114 where the verified imageresolution/code combination is replaced by the test imageresolution/half-scan code combination after which control passes toblock 116. At block 116, the test image resolution is decreased by oneand control passes back to block 62 in FIG. 3 where the processdescribed above is repeated. Referring again to decision block 108,where the worst case peak score is less than 105% of the highest otherscore within the score space outside region 201, the half-scan code isnot robust for the current test image resolution and therefore controlpasses to block 116 where the test image resolution is decreased by 1and control again passes to block 62 in FIG. 3.

Referring yet again to FIGS. 1 and 6, at block 100, where the currenttest image resolution is less than the verified image resolution dividedby the half-scan code at the test image resolution cannot be faster thanthe verified image resolution and full-scan code and therefore thehalf-scan code should not be tested for the current test imageresolution and control passes to block 102. At block 102, the currenttest image resolution is decreased by one (or other suitable value) andat block 104 processor 14 determines whether or not the next test imageresolution (e.g., the image resolution set at block 102) is greater thanthe verified image resolution divided 2. Here, an image resolution mustbe at least two times larger or coarser for a full scan to be fasterthan a quarter scan and therefore, to continue searching for an imageresolution and quarter-scan combination that is faster than a stored andverified combination including the full scan code, the condition atblock 104 must be met. Where the condition at block 104 is not met, theprocess ends because the verified and store image resolution and fullscan code are the fastest accurate and robust combination. Where thecondition at block 104 is met, control passes back to block 62 in FIG. 3the process described above continues at a finer image resolution.

The particular embodiments disclosed above are illustrative only, as theinvention may be modified and practiced in different but equivalentmanners apparent to those skilled in the art having the benefit of theteachings herein. Furthermore, no limitations are intended to thedetails of construction or design herein shown, other than as describedin the claims below. It is therefore evident that the particularembodiments disclosed above may be altered or modified and all suchvariations are considered within the scope and spirit of the invention.Accordingly, the protection sought herein is as set forth in the claimsbelow.

Thus, this disclosure is to cover all modifications, equivalents, andalternatives falling within the spirit and scope of the invention asdefined by the following appended claims. For example, while a specificsequence of steps is described above, in other embodiments the steps maytake a different order and achieve the same or similar result. Forinstance, referring to FIG. 3, the lower resolution images at block 66could be generated prior to block 64 or even prior to any of blocks 62,60, 58, 56, etc.

As another example, while the process described above is relatively fastbecause decisions are made during the process that eliminatesub-processes that cannot possibly identify more optimal results, inother less efficient embodiments all image resolution and codecombinations may be attempted to generate results and then processor 64may select the optimal image resolution/scan code combination from thefull slate of results. This process 250 is shown in FIG. 8 where, atblock 252, processor 14 identifies all resolutions that have locationerrors less than the threshold level (see again blocks 52-70 in FIG. 3).At block 254, for each identified image resolution, processor 14identifies the sparsest robust scan code. At block 256, processor 14identifies the fastest image resolution/scan code combination afterwhich the process ends.

Similarly, the first portion of the process that ends at block 70 inFIG. 3 may be performed for all resolutions supported by the system toidentify resolutions that would possibly work as recognition startingpoints and thereafter, for possibly starting point resolutions, sparsestrobust scan code may be identified. Still in other cases processor 14may try quarter-scan codes for all resolutions that may possible workand, when an image resolution and quarter-scan code combination with thecoarsest image resolution that works is identified, may only try fulland/or half-scan codes with coarser image resolutions to speed up thetraining process.

In addition, referring again to FIG. 3, while coarse subpixel phaseimages are generated at block 66, it should be appreciated that in otherembodiments the test image could be used to generate other lowerresolution images for testing purposes. For instance, in someembodiments the test image from block 53 could be translated in one ormore different degrees of freedom to generate other test images. Here,the different degrees of freedom could include rotation, shear, aspectand scale. Thereafter the testing at blocks 68 and 70 could be appliedto each of lower resolution images. As another instance, one or moredegrees of translation could be applied to the test image to generate anintermediate image set and then each intermediate image could be used togenerate a complete set of coarse subpixel phase images and the block 68and 70 process could be applied to each of the subpixel phase images.Any combination of degrees of freedom and subpixel phases arecontemplated.

In still other embodiments finding a coarsest robust test imageresolution for initiating a recognition process may be usefulindependent of whether or not a fastest scan code for the imageresolution is identified. To this end, a sub-process 260 that may besubstituted for a portion of the process shown in FIG. 3 is illustratedin FIG. 9. In sub-process 260, blocks that are similar or identical toblocks described above with respect to FIG. 3 are identified via thesame numbers and operate in the same fashion as described above unlessindicated otherwise. For instance, block 62 in FIG. 9 is identical toblock 62 in FIG. 3, block 64 in FIG. 9 is identical to block 64 in FIG.3, and so on.

Referring to FIGS. 3 and 9, after the RPF is identified at block 58 inFIG. 3, control passes to process block 60 in FIG. 9 where a test imageresolution TR is set equal to a coarsest image resolution (e.g., 25 inthe illustrated example). In FIG. 9, unlike in FIG. 3, no flag is set atblock 60 because the flag is not needed since the FIG. 9 sub-process 260stops once one robust image resolution is identified. After block 60control passes to blocks 62, 64 and 66 which operate as described abovewith respect to FIG. 3.

At block 68, the model developed at block 62 is used to examine eachlower resolution image from block 66 at full scan code to generatelocation errors in image space. At block 70, the locations for eachlower resolution image identified at block 68 are compared to thethreshold percentage (e.g., 35%) of the RPF and, where any one of thelocation errors exceeds the threshold percentage, control passes toblock 72 where TR is decremented and control passes to block 73. Atblock 73, as in FIG. 3, where TR is greater than one control passes backup to block 62 and where TR is less than one the process ends. Referringagain to block 70 in FIG. 9, where none of the location errors exceedsthe threshold percentage of the RPF, control passes to block 262 wherethe TR is stored as a verified image resolution.

Thus, sub-process 260 in FIG. 9 identifies a coarsest robust imageresolution to be used as an initial image resolution during subsequentrecognition processes for identifying a pattern with repeating elements.

In addition to being useful during a model training process for use witha pattern that includes repeating elements, it is contemplated that aversion of the process shown in FIG. 3 may also be useful during atraining process for use with general patterns that do not includerepeating elements. To this end, referring to FIG. 10, a sub-process 270that may be substituted for a portion of the process in FIG. 3 isillustrated. Here again, blocks that are identical or similar to blocksdescribed above with respect to FIG. 3 are labeled using the samenumbers and operate in the fashion described above unless indicatedotherwise.

Referring to FIGS. 3 and 10, after block 56, control passes to block 60in FIG. 10. Thus, here, because sub-process 270 is for developing amodel for a pattern that may not include repeating elements, block 58from FIG. 3 is skipped and it is assumed that some threshold (e.g., athreshold offset in pixels from a known or actual location of thepattern in the image) for a location error is instead specified andknown.

Blocks 60, 62, 64, 66 and 68 function as described above with respect toFIG. 3 and separate location errors for each lower resolution image atblock 66 are generated. Here the location errors may each indicate adifference (e.g., in pixels) between a known or actual location of thepattern in the full image resolution image and the identified locationin the lower resolution image. At block 70, the location errors for eachof the lower resolution images identified at block 68 are compared to athreshold location error or accuracy limit and, where at least onelocation error is greater than the accuracy limit, control passes toblock 65 that functions as described above with respect to FIG. 3.Referring again to block 70 in FIG. 10, where none of the locationerrors associated with the lower resolution images for the current TR isgreater than the accuracy limit, control passes to block 74 where theprocess described above with respect to FIGS. 3 and 6 continues until anoptimal TR and scan code combination is identified and stored as averified TR/code combination. Here, the accuracy limit may include anysuitable limit. For instance, in some embodiments the limit may beselected to ensure that accuracy using an associated image resolution iswithin a specific fraction of a recovery ability of a second imageresolution refinement step. As another instance, the threshold oraccuracy limit may require an accuracy to be better than the morerestrictive of a percentage of a repeating pattern frequency or therecovery ability of a second image resolution refinement step. Otherlimits are contemplated.

In other embodiments a training process similar to the process of FIGS.3 and 9 may be performed for a general pattern (e.g., a pattern that mayor may not include repeating elements) where only test resolutions aretested (e.g., where the sparsest robust scan code is not identified). Tothis end, a sub-process 280 that may be substituted for a portion ofFIG. 3 is illustrated in FIG. 11. In FIG. 11, again, blocks that aresimilar to or identical to blocks in FIG. 3 are labeled in a similarfashion and operate or function in the ways described above unlessindicated otherwise.

Referring to FIGS. 3 and 11, after block 56 in FIG. 3, control passes toblock 60 in FIG. 11 (i.e., as in the sub-process of FIG. 10, block 58 inFIG. 3 is skipped here because there may be no repeating elements in thetraining pattern). Here, as in the sub-process of FIG. 10, because theprocess stops once one robust image resolution is identified, no flag isset at block 60 (compare to block 60 in FIG. 3 where the process maycontinue after a robust image resolution is identified and therefore theflag is needed).

Blocks 62, 64, 66 and 68 function as described above with respect toFIG. 3. At block 70, the location errors for each of the lowerresolution images at the current test image resolution are compared toan accuracy limit or threshold location error and where any one of thelocation errors is greater than the accuracy limit, control passes toblocks 72 and 73 that function as described above with respect to FIG.3. At block 70, where none of the location errors is greater than theaccuracy limit, control passes to block 262 where the current TR isstored as a verified image resolution and the process ends.

From the foregoing it should be appreciated that, in a general sense,the present disclosure contemplates a generalized training process thatcould be used with any model based pattern recognition system toidentify one or more robust starting resolutions for use duringsubsequent pattern recognition processes. To this end, a generalizedprocess 300 is illustrated in FIG. 12. At block 302, test images of apattern are generated at different test resolutions. At block 304, apattern recognition model is generated for each of the test resolutions.At block 306, different lower resolution images are generated for eachimage resolution. At block 308, each model is used to examine the lowerresolution images at the associated image resolution to generatelocation errors. At block 310, at least one of the TRs that haslocations resolved for each lower resolution image within a threshold ofthe actual pattern location is stored for subsequent use.

While many of the processes described above function to identify anoptimal image resolution and scan code combination or an optimal imageresolution, it should be appreciated that similar processes arecontemplated where the selected combination or image resolution, whilerobust, may not be optimal. For instance, where five resolutions may berobust as starting points for subsequent recognition processes, a secondor third fastest of the five image resolution may be selected.

In addition, it is contemplated that in some embodiments, instead ofidentifying one robust image resolution/code combination or imageresolution, other processes may identify two or more robust combinationsor resolutions where the multiple combinations or resolutions may beused sequentially during subsequent processes when initial recognitionattempts fail. Thus, for instance, where first and second robustcombinations are identified and stored for subsequent use, if arecognition process initiated with the first combination fails, a secondrecognition process may be initiated using the second combination, andso on.

Moreover, in at least some embodiments it is contemplated that server 14may include multiple processors, processor cores or computers so thatmultiple threads can be carried out in parallel to identify robustcoarse image resolution and scan code combinations or a fastest robustcombination relatively quickly. Here, for instance, one processor maysearch for resolution/code combinations starting with a lowestresolution image and quarter code while another processor starts with alowest resolution image and full scan code and a third processor maystart with a lowest resolution image and half scan code. Other parallelprocesses are contemplated.

Referring now to Table 1 below, exemplary results of a substantiallycomplete training process as described above with respect to FIG. 3 areillustrated along with results from a traditional training algorithm.The results are arranged in columns including an image resolutioncolumn, a worst location error column, a sparsest robust scan codecolumn, a number of probes column and a traditional training ratingscolumn. In this example, the repeating pattern frequency (see block 58in FIG. 3) was 19 pixels, meaning that the worst case location error hasto be less than 6.65 pixel (i.e., 0.35×19). Assuming a minimum number oftest probes of 64, the exemplary training method would select a fastestcombination including an image with resolution of 23 and a quarter-scancode (e.g., the image resolution of 23 and quarter-scan combination hasa worst case location error 4.60 less than 6.65 and has at least theminimum number of probes). Although not shown, based on the traditionaltraining ratings, the image resolution and code combination that wouldhave been selected would have been an image resolution of 9 and thequarter-scan code. As seen in Table 1 below, an image resolution of 9requires half-scan to work properly in this example so the traditionaltraining algorithm would have selected a starting image resolution andcode combination that would not have robustly located the pattern in theimage.

TABLE 1 Worst Sparsest Traditional Location Robust Number Training Imageresolution Error Scan Code Probes Ratings 25 10.04 eScanQuarter 47 248.56 eScanQuarter 64 23 4.60 eScanQuarter 64 22 30.8 eScanQuarter 64 214.19 eScanQuarter 64 4232 20 2.92 eScanQuarter 64 6328 19 5.23eScanQuarter 64 3283 18 3.82 eScanQuarter 64 6298 17 3.44 eScanQuarter64 5819 16 5.34 eScanQuarter 64 6208 15 2.87 eScanQuarter 64 6218 142.78 eScanQuarter 64 6426 13 1.47 eScanQuarter 64   3079+ 12 1.72eScanQuarter 64 6296 11 1.04 eScanQuarter 64 6454 10 2.15 eScanQuarter67 6436 9 1.06 eScanHalf 64 6741 8 1.34 eScanFull 64 6829 7 1.69eScanFull 19 4377 6 Not Found — — — 5 Not Found — — — 4 Not Found — — —3 18.89 eScanFull 63  158 2 18.92 eScanFull 1174  877

To apprise the public of the scope of this invention, the followingclaims are made:

What is claimed is:
 1. A method for training a pattern recognition algorithm for a machine vision system that uses models of a pattern to be located, the method comprising the steps of: (a) identifying the known location of the pattern that includes repeating elements within a fine image resolution image; (b) using the fine image resolution image to train a model associated with the fine image; (c) using the model associated with the fine image to examine the fine image resolution image and to generate a score space; (d) examining the score space to identify a repeating pattern frequency; (e) using a coarse image at a coarse image resolution that is coarser than the finest image resolution image to train a model associated with the coarse image; (f) using the model associated with the coarse image to examine the coarse image thereby generating a location error; (g) comparing the location error to the repeating pattern frequency; and (h) based on the comparison, determining if the coarse image resolution is suitable for locating the pattern within a fraction of one pitch of the repeating elements.
 2. The method of claim 1 further including the step of using the coarse image to generate a plurality of lower resolution images at different coarse subpixel phases and wherein steps (f) through (h) are performed for each of the lower resolution images.
 3. The method of claim 1 further including the steps of (i) using the model associated with the coarse image to examine the coarse image thereby generating a score space associated with the coarse image and (j) examining the score space associated with the coarse image to identify a sparsest scan code that is robust at the coarse image resolution.
 4. The method of claim 3 further including the step of using the coarse image to generate a plurality of lower resolution images at different coarse subpixel phases and wherein steps (i) and (j) are performed for each of the lower resolution images.
 5. The method of claim 1 wherein the fine image resolution is a finest image resolution.
 6. The method of claim 4 wherein the fine image resolution is a finest image resolution.
 7. The method of claim 1 wherein each model includes a plurality of probes that are each a measure of similarity of at least one of a run-time image feature and a run time image region to at least one of a pattern feature and a pattern region, respectively, at a specific location.
 8. The method of claim 3 wherein each model includes a plurality of probes that are each a measure of similarity of at least one of a run-time image feature and a run time image region to at least one of a pattern feature and a pattern region, the step of using the model associated with the fine image to examine the fine image resolution image and to generate a score space including the steps of applying the plurality of probes in the model associated with the fine image at a plurality of poses to the fine image resulting in a plurality of score spaces at each pose where the score spaces include peaks, the peaks indicating likely locations of occurrences of the pattern in the fine image at each pose.
 9. The method of claim 8 wherein the step of using the model associated with the coarse image to examine the coarse image resolution image and to generate a score space includes the steps of applying the plurality of probes in the model associated with the coarse image at a plurality of poses to the coarse image resulting in a plurality of score spaces at each pose where the score spaces include peaks, the peaks indicating likely locations of occurrences of the pattern in the coarse image at each pose.
 10. An apparatus for training a pattern recognition algorithm for a machine vision system that uses models of a pattern to be located, the apparatus comprising: a processor programmed to perform the steps of: (a) identifying the known location of the pattern that includes repeating elements within a finest image resolution image; (b) using the finest image resolution image to train a model; (c) using the model to examine the finest image resolution image and to generate a score space; (d) examining the score space to identify a repeating pattern frequency; (e) using a coarse image at a coarse image resolution that is coarser than the finest image resolution image to train a model; (f) using the model associated with the coarse image to examine the coarse image thereby generating a location error; (g) comparing the location error to the repeating pattern frequency; and (h) based on the comparison, determining if the coarse image resolution is suitable for locating the pattern within a fraction of one pitch of the repeating elements.
 11. The apparatus of claim 10 wherein the processor is programmed to further perform the steps of using the coarse image to generate a plurality of lower resolution images at different coarse subpixel phases and wherein steps (f) through (h) are performed for each of the lower resolution images.
 12. The apparatus of claim 10 wherein the processor is further programmed to perform the steps of (i) using the model associated with the coarse image to examine the coarse image thereby generating a score space associated with the coarse image and (j) examining the score space associated with the coarse image to identify a sparsest scan code that is robust at the coarse image resolution.
 13. The apparatus of claim 12 wherein the processor is further programmed to perform the step of using the coarse image to generate a plurality of lower resolution images at different coarse subpixel phases and wherein steps (i) and (j) are performed for each of the lower resolution images.
 14. The apparatus of claim 10 wherein the fine image resolution is a finest image resolution.
 15. The apparatus of claim 3 wherein the fine image resolution is a finest image resolution.
 16. The apparatus of claim 10 wherein each model includes a plurality of probes that are each a measure of similarity of at least one of a run-time image feature and a run time image region to at least one of a pattern feature and a pattern region, respectively, at a specific location.
 17. The apparatus of claim 12 wherein each model includes a plurality of probes that are each a measure of similarity of at least one of a run-time image feature and a run time image region to at least one of a pattern feature and a pattern region, the step of using the model associated with the fine image to examine the fine image resolution image and to generate a score space including the steps of applying the plurality of probes in the model associated with the fine image at a plurality of poses to the fine image resulting in a plurality of score spaces at each pose where the score spaces include peaks, the peaks indicating likely locations of occurrences of the pattern in the fine image at each pose.
 18. The apparatus of claim 17 wherein the step of using the model associated with the coarse image to examine the coarse image resolution image and to generate a score space includes the steps of applying the plurality of probes in the model associated with the coarse image at a plurality of poses to the coarse image resulting in a plurality of score spaces at each pose where the score spaces include peaks, the peaks indicating likely locations of occurrences of the pattern in the coarse image at each pose. 